A semiconductor memory device with three-dimensional structure has a structure in which a memory cell array including a plurality of memory cells is integrated with a peripheral circuit. The memory cell array is provided with a stacked body in which a plurality of electrode films is stacked. Memory holes are formed in the stacked body. The end part of the stacked body is processed into a staircase shape. Each electrode film is extracted to the outside of the stacked body through a contact. In such a staircase-shaped end part, with the increase of the number of stacked electrode films, the contact may punch through on the shallow hole side. This causes the problem of poor reliability.